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Abstract 


Previous research suggests that school-based mentoring programs like those offered by Big 
Brothers Big Sisters of America (BBBSA) yield small but statistically significant improve- 
ments in the academic perfonnance of mentored students and in their beliefs in their own 
scholastic efficacy. The present study uses data from a randomized control trial involving over 
1 ,000 students from 7 1 schools across the country to investigate further the academic benefits 
of school-based mentoring, and to enrich the field’s understanding of how schools can use 
volunteers to support students. We employ instrumental variables and other approaches to 
provide insight into why the BBBSA school-based mentoring program is effective, finding that 
the relationship between mentor and protege appears to play a key role. The evidence suggests 
that developing a close relationship with a mentor led to better academic outcomes for stu- 
dents; in contrast, students who were mentored but did not experience a close relationship 
showed no improvement in academic outcomes relative to the control group. This pattern 
holds for mentoring relationships of various durations. In addition, there is no evidence that 
mentoring programs with an academic focus produced better academic outcomes than relation- 
ship-only programs. Findings do reveal, however, that programs structured with weekly 
meetings and with opportunities for pairs to interact outside of a large-group setting were more 
likely to generate close mentor-protege relationships. Beyond reporting new empirical find- 
ings, this paper contributes a theoretical structure with which to assess the results of random- 
ized evaluations of mentoring programs. 
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I. Introduction 


Financially distressed school districts need to identify low-cost ways to support struggling 
students in struggling schools. School-based mentoring programs (SBM), in which volunteers 
meet regularly with students on school grounds, are an increasingly popular option. As of 2005, 
the last date for which data are available, SBM was the fastest-growing form of mentoring in 
the United States, with 870,000 adults mentoring children in schools as part of one formal 
program or another (MENTOR, 2006). 

The mission of mentoring programs such as those offered by Big Brothers Big Sisters 
of America (BBBSA) is to “provide children facing adversity with ... one-to-one relationships 
that change their lives for the better” (Big Brothers Big Sisters of America, 2013c). Even when 
based in schools, the primary objective of the mentoring program is to provide a friend, not a 
tutor, as illustrated by the following description from the organization’s website: 

Although it takes place at schools, our School-Based Mentoring program 
isn’t limited to the classroom. Of course, some Littles do talk with their Bigs 
about class, or do homework, or read together, but it’s perfectly fine to shoot 
hoops in the gym or play on the playground. At the end of the day, it's really 
all about starting a friendship, providing guidance and inspiring them to 
reach their potential. (Big Brothers Big Sisters of America, 2013b) 

The overarching aim of this paper is to enrich the field’s understanding of how volun- 
teer mentors can best support the academic mission of schools. Our central empirical analysis 
investigates whether emotionally closer relationships between mentors and proteges lead to 
better academic outcomes. We wish to determine whether SBM works primarily through the 
connection that the protege feels with the mentor rather than through other, more direct process- 
es, such as engaging the child in academic activities during meetings. 

Prior research on mentoring relationships outside of school does point toward relation- 
ship closeness (for example, DuBois & Neville, 1997; Parra et ah, 2002) and related indicators 
of the emotional quality of the mentor-protege tie (for example, Thomson & Zand, 2010; Zand 
et ah, 2009) as important influences on youth outcomes. There is preliminary evidence that this 
may also be the case for SBM, or at least that closeness promotes protege and mentor percep- 
tions of relationship quality (Pryce & Keller, 2012). To date, however, there has not been a 
rigorous test of the role of relationship closeness in promoting academic outcomes in SBM. The 
present research addresses this gap using data from a randomized trial evaluation of the BBBSA 
SBM program conducted by Public/Private Ventures (P/PV). Because of course relationship 
closeness cannot be randomly assigned, we use instrumental variables and other controls to 
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account for factors such as the preexisting characteristics of proteges that may contribute to both 
relationship closeness and youth outcomes. 

Prior work with the same data set examined the effects on academic outcomes of match 
length and status (that is, whether the original match was intact and, if not, whether the protege 
had been rematched with a new mentor). That work found negative impacts for proteges given a 
new mentor after a prior match was terminated (Grossman, Chan, Schwartz, & Rhodes, 2012). 
Theoretically, though, a positive emotional quality or tenor in a mentoring relationship should 
mediate the benefits of program participation, independent of whatever effects may be associat- 
ed with relationship length or status (Rhodes, 2005). Accordingly, in our analyses we explore 
whether relationship quality matters even when relationship length and current status are taken 
into account. 

Finally, we investigate the characteristics of programs, mentors, and proteges that may 
increase the likelihood of a close relationship. We present findings that may help practitioners 
design SBM programs more conducive to close mentor-protege relationships, and perhaps to 
more positive academic outcomes. 

The next section provides background information on youth mentoring practice and re- 
search. The third section of the paper summarizes the experimental design and data, followed 
by a discussion of our analytical method. Section IV presents our results, and Section V offers a 
summary and discussion. 
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II. School-Based Mentoring: Background 


Mentoring is a youth development intervention that seeks to provide a child with an additional 
caring adult relationship. Mentoring programs pair children with volunteer adults or older 
students and facilitate regular meetings of the pairs. The theory is that the child benefits from 
the additional adult support and guidance. Mentoring organizations like Big Brothers Big 
Sisters target children in risky circumstances, including “children living in single parent homes, 
growing up in poverty, and coping with parental incarceration” (Big Brothers Big Sisters of 
America, 2013a). 

In the school-based form of mentoring, adults or older students meet with their proteges 
on school grounds during the school day or immediately after, typically for one hour per week 
during the academic year. BBBSA began partnering with schools over fifteen years ago to 
extend the reach of its successful community-based mentoring (CBM) services. Although 
BBBSA screens, trains, and supervises mentors in both types of programs, the contact and 
context of SBM is more constrained than in the community-based version. CBM, as practiced by 
BBBSA and other organizations, typically involves longer meetings, and locations and activities 
that the pairs choose themselves. The more limited time commitment and firmer structure of 
SBM make it easier for agencies to recruit volunteers. And because protege referrals to SBM 
programs come from teachers and school personnel rather than being dependent on parental 
initiative as in CBM, SBM also reaches groups of children who would not otherwise be served 
(Herrera, 2004). Although there are other providers of SBM, BBBSA is the largest in the United 
States. It alone administered 126,000 school-based matches in 2006 (Herrera et al., 2007). 

Available evidence on the effectiveness of SBM from randomized control trials is en- 
couraging, though somewhat mixed. The P/PV random assignment impact evaluation of the 
BBBSA SBM program that provides the data for the present research found that students who 
were assigned to receive mentoring showed small but statistically significant improvements in 
several academic outcomes, including teacher-rated academic perfomiance and student-reported 
feelings of scholastic efficacy (Herrera et ah, 2007). On the other hand, an evaluation of SBM 
programs funded by the U.S. Department of Education’s Student Mentoring Program concluded 
that the program did not have significant effects on academic outcomes (Bernstein et ah, 2009). 
In considering the results of this latter evaluation, Wheeler, Keller, and DuBois (2010) noted 
that its findings “may have been attenuated by a range of factors such as the absence of a well- 
delineated program model, a lack of additional implementation support for study agencies, and 
a relatively high percentage of the children in the intervention group who were never paired 
with a mentor.” 
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In contrast, the BBBSA SBM program has well-defined national standards. These in- 
clude, for example, an expectation that mentors receive monthly support contacts from program 
staff, a factor that research indicates is important for establishing high-quality mentoring 
relationships (Herrera, DuBois, & Grossman, 2013). Of further note is that the agencies partici- 
pating in the P/PV evaluation of the BBBSA SBM program were required to have at least four 
years of experience delivering school-based mentoring, strong agency leadership, and strong, 
established relationships with participating schools (Herrera et ah, 2007). Moreover, in this 
evaluation only 7 percent of the children in the intervention group were never matched with a 
mentor, less than half the rate in the U.S. Department of Education study. For these reasons, 
analysis of the data from the P/PV evaluation offers an especially rigorous and infonnative basis 
for examining mechanisms underlying the effects of SBM. Furthermore, agencies involved in 
this trial had the opportunity to vary potentially consequential aspects of program practice, 
including the extent to which academics were a focus during meetings between mentors and 
proteges. We exploit this variation to explore whether some program practices are more or less 
conducive to close relationships between proteges and mentors. 

Theoretical models developed in the context of CBM programs suggest that the critical 
component of mentoring is the relationship that develops between the child and the mentor 
(Rhodes, 2005). Such models typically assume that if a supportive adult consistently spends 
time with a child, the pair will fonn an emotionally close and mutually trusting relationship. 
Thanks to this relationship the child can begin to develop more positive relationships with 
teachers, parents, and peers, and begin to think more positively about him- or herself (Harter, 
1990), ultimately leading to better outcomes in areas such as academics. Available research 
backs up these theoretical models: relationship quality, including feelings of closeness specifi- 
cally, does have an important influence on youth outcomes. 

Most of this research, however, has been on mentoring outside of school. There are 
several reasons it may not automatically be possible to generalize its conclusions to SBM. For 
example, mentoring relationships are typically shorter in SBM programs than in CBM pro- 
grams. Those relationships might not have a chance to reach the levels of closeness or emotion- 
al connection needed to promote better academic outcomes (by, for example, radiating effects 
on the protege’s relationships with teachers and parents). 

On somewhat different grounds, Cavell and Elledge (forthcoming) also challenge the 
assumption that a good relationship is a precondition for positive youth outcomes, which they 
refer to as the “mentoring-as-relationship” perspective. They argue that mentoring also may be 
influential because it provides a context for activities or experiences that serve specific goals, 
such as the development of improved academic skills. From this alternative “mentoring-as- 
context’ ’ perspective, they note that benefits to proteges would be expected to accrue independ- 
ent of their mentoring relationships’ strength and duration. Because SBM programs are by 
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definition based in schools, and because some SBM programs specifically emphasize academic 
activities, mentoring-as-context mechanisms could end up accounting for the academic benefits 
of program participation. The present investigation is thus well suited to inform the debate on 
this important, unresolved issue. 

In their investigation using data from the same P/PV study, Grossman et al. (2012) find 
little evidence that program impacts on academic outcomes varied according to the length of 
time a protege was matched with a mentor, but they do find variation among proteges based on 
match status. Relative to controls, proteges in intact mentoring relationships showed significant 
improvement on teacher-rated academic performance. On the other hand, those who had been 
rematched with a new mentor after an initial match ended did significantly worse than the 
controls. These negative effects were not observed for proteges whose initial mentoring rela- 
tionships had ended but who had not been rematched. Similarly, despite the findings of Gross- 
man et al., prior research with CBM programs has supported the hypothesis that mentoring 
relationships and their effects grow stronger over time. As we investigate whether relationship 
closeness is the key ingredient to an effective school-based mentoring experience, we therefore 
take care to separate the effects of closeness from the potentially confounding effects of both 
match length and match status. 

It is also useful to consider briefly the evidence on a broader array of school-based pre- 
vention programs, as they too introduce caring adults into students’ lives. A meta-analysis of 
evaluations of social and emotional learning programs in schools (Durlak et al., 2011) found that 
program effects on academic and other outcomes were substantially greater when programs were 
structured to engage students in systematic skill-building activities (that is, “SAFE”: Sequenced, 
Active, Focused, and Explicit). On the surface, this finding would seem to suggest that a close 
relationship with a mentor is not necessarily the key to better academic outcomes. Yet SBM 
programs specifically focus on cultivating emotional connections between children and adults, so 
they may capitalize on relational factors to foster academic improvements among participants. 
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III. Method 


Study Design and Sample 

The sample for our study consists of the students who participated in the randomized control 
trial of the Big Brothers Big Sisters of America (BBBSA) SBM program during the 2004-2005 
school year (Herrera et al., 2007). Study participants were recruited by 10 BBBSA study 
agencies across the country, each with four or more years of experience in SBM. Recruitment 
proceeded in the usual way, that is, mostly through referrals by school staff; children had to be 
entering grades four through nine and had to have parental consent to be part of the research. 
Over a thousand (1,139) students across 71 public schools in rural and urban school districts (41 
elementary, 27 middle, and 3 high schools) met these criteria. After completing a baseline 
survey, half (565) of these students were randomly assigned by researchers to the treatment 
group to be matched with a mentor and the other half (574) were placed on a waiting list to be 
matched 15 months later, at the end of the study. 1 Randomization was stratified by school. The 
students’ teachers and mentors also completed baseline surveys in the fall of 2004. Follow-up 
surveying of all groups occurred at the end of the school year, beginning in April 2005. 

Table 1 reports basic descriptive statistics for the control and treatment samples at base- 
line. Overall, just over half (54 percent) of the students were female and the majority (63 
percent) were members of ethnic and racial minority groups. Most (69 percent) received free or 
reduced-price lunches. On average they struggled academically, receiving below a 3.0 on the 
teacher survey (the rating for “average” or “satisfactory” performance). The average age was 
11. As shown in Table 1, there were no statistically significant differences between the control 
group and the group assigned to treatment, indicating successful randomization. 

All but 6 percent of the proteges completed the survey at follow up. Survey completion 
rates were slightly lower for teachers, with follow-up outcome measures missing for approxi- 
mately 16 percent of the overall sample. 2 Importantly, however, survey noncompletion rates are 
almost identical for the treatment and control groups (6 percent versus 7 percent for the student 
survey, and 16 percent versus 17 percent for the teacher survey). According to evidence 
standards developed and used by the What Works Clearinghouse of the U.S. Department of 
Education’s Institute of Education Sciences (2011), this combination of overall and differential 
attrition should result in an acceptable level of bias even under conservative assumptions. 


'There were originally 575 control students but one dropped out of the study. Students who were referred 
by Child Protective Services were exempt from the study lottery but were not included in the study. 

‘Analyses indicate that students with lower baseline academic performance were slightly less likely to 
complete the survey, so the baseline academic measure is included as a covariate in all specifications. 
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Table 1 


Student Characteristics at Baseline 



Control 


Treatment 

Difference 

(Control- 

Treatment) 


Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

Mean 

Standard 

error 

Age 

11.22 

1.66 

11.24 

1.67 

-.02 

.099 

African-American/Hispanic 

(proportion) 

0.61 

0.49 

0.64 

0.48 

-.04 

.029 

Female (proportion) 

0.54 

0.50 

0.54 

0.50 

.00 

.030 

Free/reduced-price lunch (proportion) 
Teacher-rated outcome measures: 

0.69 

0.43 

0.69 

0.43 

-.00 

.025 

Overall academic performance 

2.47 

1.09 

2.56 

1.10 

-.08 

.070 

Quality of schoolwork 

2.82 

0.91 

2.84 

0.94 

-.02 

.058 

Completion of schoolwork 
Student-reported outcome measure: 

2.96 

0.99 

3.00 

1.06 

-.04 

.065 

Scholastic efficacy 

2.75 

0.64 

2.80 

0.62 

-.05 

.038 

Sample size 

574 


565 





NOTES: See Section III, “Outcome Measures and Key Variables,” for an explanation of how outcome 
measures were derived. A two-tailed t-test was applied to differences between the groups. No differences 
were statistically significant at a 10 percent level. 


Because SBM does not require same-gender matches (most CBM programs do), the 
percentage of female mentors was greater than the percentage of female students. In fact, 
mentors were predominately female (72 percent). More than three-quarters (77 percent) of the 
mentors were white, and nearly half (48 percent) were high school students. 


Program Overview 

About half (49 percent) of SBM programs had proteges meet their mentors only during the 
school day, with the others meeting after school (47 percent) or both during and after school (4 
percent). Programs reported using a variety of locations in the school for meetings, including 
the cafeteria (41 percent), library (34 percent), and a designated classroom (33 percent). Some 
programs used more than one meeting place. Almost half of all mentoring pairs shared their 
meeting space with other pairs. 

Table 2 presents some statistics describing the features of the programs and characteris- 
tics of the mentors in this study. These variables are used in the analyses presented in Section 
IV, “Facilitating Relationships in School-Based Mentoring,” which explore the characteristics 
of programs and mentors that increase the likelihood of a close relationship. Mentors reported 
engaging in a wide variety of activities with their proteges, most of which were chosen by 





protege and mentor together (49 percent) or by the protege alone (20 percent). For example, 
mentors reported doing each of the following types of activities “a lot”: talking casually (71 
percent), talking about family or friends (43 percent and 44 percent, respectively), talking about 
the future (30 percent), playing indoor games (54 percent), doing creative activities (36 percent), 
playing sports (25 percent), doing homework (27 percent), and talking about academic issues 
(3 1 percent). Many pairs met for between 45 and 60 minutes (40 percent) but another 39 percent 
met for more than an hour. Almost all matches met three (10 percent) or four (79 percent) times 
a month. Over the year, the average protege received 17 hours of mentoring over 5.3 months. 

On average, the BBBSA agencies overseeing the school programs had 9.5 years of ex- 
perience implementing SBM. BBBSA supported sites during implementation with meetings 
and teleconferences. Prematch mentor training lasted about 45 minutes on average. 


Table 2 

Program Features 


Treatment 

Mean 

Standard Deviation 

Academically focused program 3 

0.40 

0.49 

All pairs meeting at same time and location 3 

0.44 

0.50 

Some structured time 3 

0.71 

0.46 

Infrequent meetings 3 

0.13 

0.33 

Volunteer recognition 11 

2.63 

1.49 

High school student mentor 3 

0.49 

0.50 

College student mentor 3 

0.16 

0.37 

Same gender mentor as student 3 

0.80 

0.39 

Same race mentor as student 3 

0.49 

0.50 

“Mentor shows up when he/she is supposed to” c 

3.33 

0.94 

Amount of individual mentor training 1 ' 

3.10 

1.45 

Sample size 

565 



NOTES: “The mean of this variable represents the proportion of students assigned to treatment who were 
mentored under the specified condition. 

b This variable records the number of ways programs recognized volunteers’ efforts. 

“The mean of this variable represents the average rating by the student on a scale from 1 = “not at all 
true” to 5 = “very true.” 

d This variable is the sum of mentor training before and after match using the following categorical 
scale: 1 = less than 30 minutes, 2 = 30 minutes to less than one hour, 3 = one hour to less than two hours, 
4 = two hours to less than 4 hours, and 5 = four hours or more. 
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Outcome Measures and Key Variables 

This paper examines changes in four academic measures, each measured at baseline and at 
follow-up for each student: 

• Overall academic performance is a rating by the student’s teacher of the 
student’s “overall academic performance” on a single- item, five-point scale 
from 1 = “below grade level” to 5 = “excellent” (Pierce, Hamm, & Vandell, 

1999). 

• Quality of work is a three-item scale score calculated as a mean of teacher 
ratings of the correctness, neatness, and completeness of a child’s classwork 
on a five-point scale from 1 = “well below average” to 5 = “well above aver- 
age” (Herrera, 2004; alpha values of 0.88 at baseline and 0.90 at follow-up 
indicate internal reliability (Cronbach, 1951)). 

• Completion of schoolwork is a two-item average of the teacher’s report of 
how often the student completes in-class and homework assignments on a 
five-point scale ranging from 1 = “well below average” to 5 = “well above 
average” (Herrera, 2004; alpha values of 0.93 at baseline and 0.94 at follow- 
up indicate internal reliability). 

• Scholastic efficacy is the average across six items of a student’s ratings of 
his or her own ability to do schoolwork on a four-point scale from 1 = “not at 
all true” to 4 = “very true.” Items include “I do very well at my class work” 
and a reverse coding of “I often forget what I learn” (Harter, 1985; alpha val- 
ues of 0.70 at baseline and 0.72 at follow-up indicate internal reliability). 

In all cases, a higher score indicates better outcomes. 

Four explanatory variables are central to our exploration of the mechanism of SBM: 


• Closeness is a single-item measure in which proteges were asked to respond 
to the question “How close do you feel to your mentor?” on a four-point 
scale where 1 = “not close at all,” 2 = “not very close,” 3 = “somewhat 
close,” and 4 = “very close.” This question was part of the follow-up survey 
completed by proteges at the end of the school year, with students instructed 
to rate their current or most recent mentor. 

• Match status is the status of the protege’s original mentoring match at the 
end of the school year. The original mentoring relationship was “intact,” or 
that relationship ended and the student experienced a “rematch” with a new 
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mentor, or the relationship ended and the student was given “no rematch” 
with a new mentor. 

• Match length is the total number of weeks of a protege’s current or most re- 
cent mentoring match (corresponding to the relationship for which we have 
closeness ratings) at the time of the follow-up survey. 3 

• Teacher relationship is the average across 1 1 items of a student’s ratings of 
teacher comiectedness (Karcher, 2003) and teacher relationship quality. 
Items are rated on a four-point scale from 1 = “not at all true” to 4 = “very 
true,” and include “I always try hard to earn my teachers’ trust” and a reverse 
coding of “My teachers don’t know me very well this year” (alpha values of 
0.82 at baseline and 0.82 at follow-up indicate internal reliability). We use 
the baseline value of this variable as a measure of the child’s ability to have a 
close relationship with a nonparental adult. 


Analysis Plan 

Regression specifications in which the outcome, Y;, is regressed on the assigned treatment 
indicator, Z;, provide unbiased estimates of the average impact of SBM across all proteges. 
Consider: 


Yi - cq + PittZ; + X;Yi + 8; (1) 

where X, is a vector of covariates, which in our analyses is the pretreatment value of the 
outcome, and S; is the unexplained variation in outcomes. These intention-to-treat (ITT) esti- 
mates, Pitt, help inform us about the average effectiveness of the programs as currently 
configured. 

Pitt provides a fairly good estimate of the impact of mentoring-as-context, since only 7 
percent of the treatment students were never matched. But because only 75 percent of the 
students assigned to treatment rated their relationship with their mentor as “somewhat close” or 
“very close,” it does not represent the impact of providing a child with a close, caring relation- 
ship with a nonparental adult, widely viewed as the essence of mentoring. The “partial compli- 
ance” of treatment students to fonning a close relationship means that Pitt does not equal the 
effect of the “close adult relationship” treatment. The randomized assignment, Z;, only influ- 


3 Grossman et al. (2012) use a definition of match length that sums all days of mentoring across all match- 
es. We define match length slightly differently, given our focus on the relationship between the two individuals 
in the match. 
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ences the probability that a student in the mentoring group receives the treatment of interest, T b 
where T, represents a close mentor-protege relationship. 4 

We therefore estimate the causal effect of having a close relationship with a mentor, 
that is, the effect among the “compliers” with the SBM treatments, a la Angrist, Imbens, and 
Rubin (1996). Regressions in which outcomes are regressed on the receipt of the intended 
treatment, T,, provide the treatment-on-treated (TOT) estimates, |3tot. 


Y ; = a 2 + PtotT; + X lT 2 + 8; (2) 

With an unbiased estimate of [3 T ot, we can gauge the impact of SBM when it is operat- 
ing as designed, providing children with close relationships with program mentors. 

In producing our estimates, we must take care not to reintroduce selection bias into the 
results. Although treatment status, Z h was assigned randomly, the students who actually 
received the intended treatment, T,, are not necessarily a random subset. These students may 
have an underlying characteristic, A,, affecting both their ability to form a relationship with a 
mentor and their academic progress over the study year — for example, a general ability to have 
good relationships with adults. One approach is to control for the characteristic explicitly. The 
youth survey includes questions about a student’s relationship with his or her teacher, and we 
use baseline responses to construct a variable A;, which provides a direct measure of a student’s 
ability to have a close relationship with a nonparental adult, to include in our ordinary least 
squares (OLS) regressions. 


Yj - a 3 + |Y,T i + pA; + X;y 3 + 8; (3) 

Alternatively, when compliance is imperfect, we can use the randomized treatment sta- 
tus, Z;, as an instrument for the treatment of interest, having a relationship with a mentor, T;. As 
established in Angrist, Imbens, and Rubin (1996), random assigmnent status is an ideal candi- 
date for an instrument because it is independent of student characteristics. We thus derive the 
Wald estimate of the causal effect among compliers: the difference in outcomes between the 
nominal treatment and control groups divided by the difference in the probability of receiving 
treatment between the groups. This estimate, which provides the effect of the treatment on the 
treated or, more generally, the local average treatment effect, measures the effect of program 
participation on those proteges who successfully comiected with their assigned mentors (Bloom, 
1984; Imbens & Angrist, 1994). Specifically, 


4 Although we know that E[Y; (0)|Z = 1] - E[Y; (0)|Z = 0] is equal to zero due to random assignment, the 
intention to treat estimate, E[Y, |Z — 1] — E[Y; |Z = 0], is equal to the causal effect of Z, not the effect of the 

treatment, T. 
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( 4 ) 


(E[Y; |Z = 1] - E[Y ; |Z = 0]) / Pr[T; =1|Z = 1] 

Two-stage least squares (2SLS) is used to estimate this effect of the treatment on the 
treated, where Z; acts as an instrument for T,. 

This instrumental variables estimate can be interpreted as the average treatment effect 
for the compliers — that is, the effect on the students who fonn a close relationship with a 
program mentor — under two assumptions: monotonicity and independence (Angrist, Imbens, 
& Rubin, 1996). Monotonicity requires that the assignment of a mentor increase the likelihood 
of a new close mentoring relationship for each and every student assigned to treatment. This 
requirement is met, since control students were not offered the opportunity to form relationships 
with BBBSA mentors. 

Independence requires two things: first, that we be able to identify the causal impact of 
the instrument, which is also true by construction here, and second, that potential outcomes not 
be directly affected by the instrument. This last requirement, also known as an exclusion 
restriction, does not necessarily hold in the case of SBM. We must therefore test the validity of 
this assumption empirically. 

In other words, we must provide evidence that the outcomes of students who met with 
mentors but did not form close relationships were not affected by mentoring. Assignment to the 
mentoring group, which is our instrument Z,, may affect students who do not form close 
relationships if, say, SBM yields academic benefits as a result of the time those students spend 
working on homework and other academic activities with mentors. We thus propose and test a 
competing hypothesis to that which presumes the centrality of relationships: namely, that 
proteges’ academic outcomes are affected by the time spent on academic activities in match 
meetings, whether or not the mentor-protege relationship is a close one. 5 

The SBM programs in this data set differ by academic intensity. Although most devote 
little meeting time to academics, 225 of 565 treatment students were in programs that reported 
spending more than 25 percent of their meeting time on academic activities such as homework 
help and tutoring. We define such programs to be “academically focused,” in contrast to the 
other, “relationship-only” programs. This variation in program focus endows the randomized 
evaluation with a cross-cutting design, in which two different treatments are tested simultane- 
ously and in which there are three assignment groups (pure controls, students mentored in an 
SBM program with an academic focus, and students mentored in an SBM without such a 
focus). This design allows us to test the validity of the exclusion assumption in our instrumental 


5 Note that these two “competing” processes are not mutually exclusive. In fact, the presence of a close 
relationship between mentor and protege may enable or intensify the effects of time spent on academic 
activities. 
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variable analyses. It also allows us to explore the effects of the interventions and their combina- 
tion relative to the control group and relative to each other. 

The following section reports the results of our empirical work. We first present evi- 
dence on the frequency and nature of close relationships in BBBSA SBM programs, as well as 
support for a simple characterization of relationships in this context. Next, we examine the 
evidence for academic benefits of SBM and report the effects of assignment to the mentoring 
treatment. Alongside the intention-to-treat estimates, we present various estimates of the effect 
of the treatment on the treated, using the approach outlined above, in order to determine if 
relationship closeness makes a difference. Third, we explore whether relationship quality still 
matters once relationship length and status are taken into account, and vice versa. We build 
upon the findings of previous analyses documenting the effects of match length and match 
status in school-based mentoring (Grossman et al., 2012) to refine the idea of “length plus 
strength” in SBM. Finally, we use logistic regression to investigate those features of SBM 
programs that may make close relationships more likely. Using the sample of all students 
assigned to treatment, we identify the program features and mentor characteristics associated 
with protege reports of close mentoring relationships. We wish to provide guidance to SBM 
program administrators so that they can deliver the treatment of close relationships in more 
effective ways. 
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IV. Results 


The Nature of Relationships in School-Based Mentoring 

Many of the SBM matches were “successes,” in that proteges characterized them as close. Of 
the 565 students assigned to treatment, 75.4 percent rated their relationships with their mentors 
as somewhat or very close at the time of the follow-up survey. The remaining 24.6 percent of 
treatment students were either never matched or rated their relationship as “not close at all” or 
“not very close.” 

Figure 1 reports the proportion of students who described their relationships with their 
mentors as somewhat or very close, by duration in months. A majority of matches of any 
duration are rated favorably by the proteges. After three months of meetings, over four-fifths of 
relationships are close, and that proportion does not increase appreciably with match duration 
— suggesting that most matches that ever achieved some degree of a close emotional connec- 
tion did so within three months. 

In addition, our initial analyses (not presented here; available upon request) suggest a 
threshold for closeness levels in SBM relationships. This threshold must be reached for a 
relationship to improve outcomes, but additional closeness beyond the threshold level has 
insignificant effects. We estimated the effect of relationship closeness on academic outcomes 
using detailed closeness ratings. Matches were categorized into four distinct groups: not close at 
all, not very close, somewhat close, and very close. We tested equality restrictions on the 
coefficients and found that mentoring experiences receiving either of the two lower closeness 
ratings produced statistically similar impacts, as did mentoring experiences receiving either of 
the two higher ratings (for overall academic performance, F(2, 937) = 0.00, p = 0.9992). These 
results support using a dichotomous measure of closeness in which relationships are recorded as 
“close” if and only if proteges rated them as “somewhat close” or “very close.” The analyses 
reported in the following sections use this measure, modeling academic performance as a 
discontinuous function of closeness. 


Impact of Relationship Quality on Academic Outcomes 

Table 3 presents our main results on the academic benefits of school-based mentoring pro- 
grams. It reports both the effects of assignment to the mentoring treatment and estimates of the 
role of relationships in SBM. The table reports impact coefficients and standard errors for each 
of the four academic outcome measures. Estimates reported as statistically significant in the 
table remain significant at a target level of significance of 0.05 after correcting for multiple 
comparisons (Benjamini & Hochberg, 1995). 
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Figure 1 


Proportion of Matches with High Protege 
Closeness Rating, by Duration in Months 
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Column (1) contains the OLS impact coefficients and standard errors from the inten- 
tion- to-treat specification presented in equation (1). Consistent with the findings reported by 
Herrera et al. (2007), we find that students who were assigned to receive mentoring show 
statistically significant improvements in their academic perfomiance and scholastic efficacy. 
Column (2) presents the potentially biased but straightforward treatment-on-treated estimates 
from OLS regression of outcomes on the treatment of interest, T„ a close mentor-protege 
relationship, as in equation (2). The estimates in column (2) are about 20 percent larger than 
those in column (1), as would be expected under the incomplete compliance with treatment. The 
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Table 3 


Estimated Impact of a Close Relationship on Academic Outcomes 



(1) 

ITT 

(2) 

TOT 

(3) 

TOT controlling 
for relationship 
ability 

(4) 

2SLS 

Overall academic performance (sample 

size = 945) 



Assigned to treatment 

Received treatment 
(had a close relationship) 

0.12** 

(0.05) 

0 

(0.05) 

Q 14*** 

(0.05) 

0.14** 

(0.06) 

R 1 

0.534 

0.535 

0.536 

0.535 

Quality of work (sample size = 

953) 




Assigned to treatment 

Received treatment 
(had a close relationship) 

0.12*** 

(0.04) 

Q 14*** 

(0.04) 

0 i4*** 

(0.04) 

0.15*** 

(0.05) 

R 1 

0.478 

0.480 

0.483 

0.480 

Completion of schoolwork (sample size 

= 953) 



Assigned to treatment 

Received treatment 
(had a close relationship) 

0.15*** 

(0.05) 

q i7*** 

(0.05) 

q i7*** 

(0.05) 

0.18*** 

(0.06) 

R 1 

0.385 

0.386 

0.390 

0.386 

Scholastic efficacy (sample size = 1067) 

Assigned to treatment 

Received treatment 
(had a close relationship) 

0.07** 

(0.03) 

0.08** 

(0.03) 

0.07** 

(0.03) 

0.08** 

(0.04) 

R- 

0.268 

0.269 

0.275 

0.269 


NOTES: This table reports impact coefficients representing the estimated absolute change in the 
outcome measure associated with (assignment to) treatment. See Section III, “Outcome 
Measures and Key Variables,” for an explanation of how outcome measures were derived. Col- 
umn (1) presents the intention-to-treat (ITT) coefficients obtained from estimating equation (1). 
Column (2) presents the treatment-on-treated (TOT) coefficients obtained from estimating equa- 
tion (2). Column (3) presents the treatment-on-treated (TOT) coefficients obtained from estimat- 
ing equation (3), which includes a variable controlling for the quality of the protege’s relation- 
ship with his or her teacher. Column (4) presents the two-stage least squares (2SLS) estimates 
presented in equation (4). Standard errors are shown in parentheses. Statistical significance lev- 
els are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 
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results indicate that a relationship rated as somewhat or very close by the protege is associated 
with better academic outcomes. 

In the next two columns, we attempt to leam more about causality. Does a close men- 
toring relationship cause the better academic outcomes? Our OLS estimates of the treatment- 
on-treated effect (column 2) may be biased due to the presence of a relevant but omitted 
variable. For instance, some children are better at working with adults, and these children may 
have both better mentoring relationships and better school performance. 

Column (3), which corresponds to equation (3), addresses the potential omitted variable 
problem by including in our OLS specifications a variable that attempts to control directly for 
the possible confounding factor — namely, the quality of the child’s relationship with his or her 
teacher before the mentoring program. This variable provides a measure of the child’s ability to 
have a close relationship with a nonparental adult, and, as expected, has a positive and statisti- 
cally significant effect on academic outcomes. As can be seen by comparing columns (2) and 
(3), including this teacher relationship variable does not affect our estimate of the impact of a 
close mentoring relationship. 

Results when applying instrumental variable methods are shown in column (4). The es- 
timated effects of a mentor-protege connection in the 2SLS model are at least as large as the 
OLS estimates presented in column (2) and indicate that there are statistically significant 
academic benefits to a close mentoring relationship. We test to ensure that our instrument is not 
weakly correlated with the relationship variable after controlling for exogenous regressors, as 
weak instruments can produce estimates that are unreliable and biased toward the OLS estimate 
(Bound, Jaeger, & Baker, 1995). We find, as expected, that assignment to the mentoring group 
strongly predicts (F = 1956, p = 0.00) a comiection with a program mentor. If we did have weak 
instruments, the 2SLS standard errors would be large and the validity of our instrument would 
be even more important, as even a slight correlation between error in the main equation and the 
instrument would lead to inconsistent estimates. 6 

Thus we find evidence that a close mentoring relationship positively affects academic 
performance. Effect sizes, obtained by dividing the impact coefficients reported in the table by 
the standard deviation of the appropriate outcome measure, range from 0.13 standard deviations 
(for overall academic perfonnance and scholastic efficacy) to 0.18 standard deviations (for 
completeness of schoolwork), and are consistent across alternative specifications. The results 
presented in columns (3) and (4) suggest that there may not be an omitted variable bias issue in 
the OLS specification in column (2), removing the need for instrumental variable techniques in 


6 Tests of overidentifying restrictions for the model, although limited, do not suggest that the instruments 
are correlated with the error term in the main equation for any outcome. 
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follow-up analyses. The OLS and 2SLS estimates of the effect of close relationships are 
remarkably similar across specifications (2) through (4). A heteroskedasticity-robust Hausman 
(1978) test of endogeneity does not reject the null hypothesis that relationship closeness is 
exogenous, and the OLS specifications appear to provide unbiased estimates of the impact of 
providing a child with a close, caring relationship with a program mentor. 

Table 4 provides further indication that these estimates can be interpreted as the causal 
effect of a close mentoring relationship. When we include a variable in our OLS regressions 
indicating the presence of a mentoring relationship that is “not close at all” or “not very close,” 
we find no effect of mentoring for pairs that are not close. If the effects of “mentored but did not 
receive treatment (did not have a close relationship)” were statistically significant, we might 
conclude that SBM enhances student academic performance through alternative pathways. 
Instead, we find no evidence that mentoring enhances academic performance if the protege does 
not have a good relationship with the mentor. 


Table 4 


Estimated Impact of Mentoring, With and Without a Close Relationship 



Overall 

Quality 

Completion 

Scholastic 


academic 

of 

of school- 

efficacy 


performance 

work 

work 


Received treatment 

0 

0 14*H=H= 

0.18*** 

0.07** 

(had a close relationship) 

(0.05) 

(0.05) 

(0.06) 

(0.03) 

Mentored but did not receive treatment 

0.03 

-0.04 

0.05 

-0.02 

(did not have a close relationship) 

(0.09) 

(0.08) 

(0.08) 

(0.06) 

Sample size 

945 

953 

953 

1067 

R 2 

0.535 

0.480 

0.386 

0.269 


NOTES: This table reports impact coefficients representing estimated absolute changes in the outcome 
measures. See Section III, “Outcome Measures and Key Variables,” for an explanation of how outcome 
measures were derived. Standard errors are shown in parentheses. Statistical significance levels are 
indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 


We next use existing variation in program structure to explore explicitly the possibility 
that mentoring affects academic performance through additional time spent studying with a 
mentor rather than through the formation of a close mentor-protege relationship. Some students 
were assigned mentors in relationship-only programs, whereas others were mentored in aca- 
demically focused programs. In each type of program, some students connected with their 
assigned mentors and thus received treatment, and others did not. Relationship-formation rates 
are essentially identical across program types. Of the 225 students assigned to treatment in 
academically focused programs, 75.1 percent comiected with their mentors by the time of the 
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follow up survey; in relationship-only programs, 75.6 percent of proteges reported feeling close 
to a mentor. 

The specifications in Table 5 are like those in columns (1) and (4) of Table 3 but they 
also include an indicator variable specifying whether or not a student was at a site that ran an 
academically focused program, and two treatment dummies: one indicating students assigned to 
the treatment group at a relationship-only site and one indicating students assigned to treatment 
at an academically focused site. In columns (1) and (3), we constrain the impacts across these 
two types of sites to be the same, whereas in columns (2) and (4) we allow them to differ and 
test whether the constraint is valid. 

The first two columns of Table 5 present intention-to-treat estimates for the constrained 
and unconstrained specifications. In general, assignment to the treatment group appears to have 
similar effects whether the mentoring occurs with an academic focus or not. For overall aca- 
demic performance, the effects of the two different treatment conditions are virtually identical. 
In completion of work students mentored in an academically focused program perform worse, 
while in scholastic efficacy they perform better. Indeed, the effects are not significantly different 
across program types for all four outcome measures. 7 Thus we see no evidence that students 
mentored in an academically focused program do better academically than students in relation- 
ship-only programs. 

We also delve more deeply to ascertain the differences in treatment effects on the treat- 
ed, though we do not expect a different verdict, given the similar rates of relationship formation 
in each program type. Columns (3) and (4) of Table 5 present the 2SLS treatment-on-treated 
estimates for the constrained and unconstrained specifications, respectively. Here we use an 
expanded instrument set created by interacting treatment assignment with program type, which 
allows us to disentangle the effects of relationship and direct academic help since the two 
instruments may have differential effects on these pathways. Column (3) uses one endogenous 
regressor and thus is overidentified. Column (4) has two endogenous regressors (our relation- 
ship closeness variable in each of the two program types) and thus is just-identified. 

We find that an academic focus does not produce a detectably different effect of having 
a close SBM relationship. Following Angrist (1991), we use the overidentification test statistic 
in column (3) to test whether the 2SLS estimates of the effects of the two program types are 
equal in column (4). We find they are not significantly different (all p-values are greater than 


7 Note that the data may not have sufficient power to distinguish the effects of the programs. Particularly in 
the cases of completion of schoolwork and of student-reported scholastic efficacy, failure to reject equality may 
be a type II error. Both also fail to reject a null hypothesis of a quarter standard deviation difference in effects, 
but the effects are in opposite directions. 
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Table 5 


Estimated Impact on Academic Outcomes, by Program Type 



(1) 

ITT 

(2) 

ITT 

(3) 

2SLS 

(4) 

2SLS 

Overall academic performance 

Assigned to treatment 
... in an RO program 
... in an AF program 

0.12** 

(0.05) 

0.12* 

(0.06) 

0.12 

(0.08) 




Received treatment (close relationship) 0.15** 

(0.06) 


... in an RO program 
... in an AF program 



0.15* 

(0.08) 

0.15 

(0.10) 

Overidentification test j“(l) 


0.000 

(p = 0.990) 


Quality of work 

Assigned to treatment 0.12*** 

(0.04) 

... in an RO program 
... in an AF program 
Received treatment (close relationship) 

... in an RO program 
... in an AF program 

0.10* 

(0.05) 

0.15* 

(0.08) 

0.15*** 

(0.05) 

0.12* 

(0.06) 

0.18* 

(0.10) 

Overidentification test j“(l) 


0.260 

(p = 0.610) 


Completion of schoolwork 

Assigned to treatment 0.15*** 

(0.05) 

... in an RO program 
... in an AF program 
Received treatment (close relationship) 

... in an RO program 
... in an AF program 

0.18*** 

(0.07) 

0.10 

(0.09) 

0.18*** 

(0.06) 

0.22*** 

(0.08) 

0.12 

(0.11) 

Overidentification test j“(l) 


0.441 

(p = 0.507) 



(continued) 


21 







Table 5 (continued) 


Scholastic efficacy 

Assigned to treatment 0.07** 

(0.03) 

... in an RO program 0.03 

(0.04) 

... in an AF program 0.14** 

(0.06) 

Received treatment (close relationship) 0.09** 

(0.04) 

. . .in an RO program 0.04 

(0.05) 

... in an AF program 0.17** 

(0.07) 

Overidentification test y 2 (l) 2.39 

(p = 0.122) 

NOTES: This table reports impact coefficients representing the estimated absolute change in the outcome 
measure associated with (assignment to) treatment. See Section III, “Outcome Measures and Key 
Variables,” for an explanation of how outcome measures were derived. 

AF = academically focused, RO = relationship-only, ITT = intention-to-treat, and 2SLS = two-stage 
least squares. 

Columns (1) and (3) constrain the impacts across AF and RO sites to be the same, whereas in columns 
(2) and (4) we allow them to differ. 

The overidentification test statistic does not reject the hypothesis that effects of the two program types 
are equal (Angrist, 1991). 

Standard errors are shown in parentheses. Statistical significance levels are indicated as: * = 10 percent; ** 
= 5 percent; and *** = 1 percent. 


0.10), and thus we find no evidence of a causal link between assigned treatment status and the 
outcomes other than the close relationship formed between mentor and protege. 


Relationship Quality vs. Relationship Length and Status 

In this section, we collate our findings with those of previous studies on the effects of mentoring 
match length and status. Innovations developed in this paper clarify previously published results 
about effective mentoring practice. 

In Table 6, we first reproduce the apparent benefits of intactness, originally reported by 
Grossman et al. (2012), for each outcome measure. When we interact the match closeness 
indicator with match status, however, we find that only close, intact matches have an impact. 
Proteges in intact matches who are not close to their mentors fared no better than they would 
have without mentors. Proteges who were rematched and had a close relationship with their 
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Table 6 


Estimated Impact of Match Status, Controlling for Relationship Closeness 



Overall academic performance 

Quality of work 

Completion of schoolwork 

Scholastic efficacy 

Intact match 

0.11** 

0.11** 

0.15*** 

0.07** 


(0.05) 

(0.05) 

(0.05) 

(0.03) 

Rematch 

0.08 

0.14 

0.15 

0.03 


(0.10) 

(0.09) 

(0.12) 

(0.07) 

No rematch 

0.06 

0.14 

0.01 

0.09 


(0.14) 

(0.10) 

(0.15) 

(0.08) 

Not close/intact match 

-0.01 

-0.08 

0.07 

-0.02 


(0.11) 

(0.10) 

(0.10) 

(0.07) 

Not close/rematch 

0.03 

-0.08 

-0.11 

-0.06 


(0.27) 

(0.26) 

(0.23) 

(0.11) 

Not close/no rematch 

-0.03 

0.16 

0.03 

0.15 


(0.21) 

(0.16) 

(0.22) 

(0.15) 

Close/intact match 

0.13** 

0.12** 

0 J7*H=H= 

0.08** 


(0.06) 

(0.05) 

(0.06) 

(0.04) 

Close/rematch 

0.09 

0.18** 

0.21* 

0.04 


(0.10) 

(0.09) 

(0.13) 

(0.08) 

Close/no rematch 

0.24 

0.21 

0.17 

0.02 


(0.20) 

(0.14) 

(0.24) 

(0.09) 

R- 

0.533 0.534 

0.478 0.480 

0.385 0.387 

0.268 0.270 


NOTES: This table reports impact coefficients representing estimated absolute changes in the outcome measures. See Section III, “Outcome 
Measures and Key Variables,” for an explanation of how outcome measures were derived. Standard errors are shown in parentheses. Statistical 
significance levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 



new mentors were not negatively affected. Indeed, there is evidence that rematched students in 
close relationships were doing significantly better than their control peers with respect to the 
quality of their schoolwork, and it is possible that the positive impacts of being matched had not 
yet grown large enough to be statistically significant for the rematched students’ other out- 
comes. Thus, the current study refines the work of Grossman et al. (2012) to suggest that 
rematching does not have negative effects as long as the new relationship clicks. 

In Table 7, we first provide analyses that include indicators of match duration but not 
closeness and reproduce the stylized fact that longer matches yield greater benefits. Once we 
control for match closeness by interacting the match closeness indicator with match duration, 
however, we observe that only close relationships have impacts and that match length does not 
matter much. In other words, if a match is long but not good, it is not productive; if it is short 
but good, it is productive. 

Our results — that longer matches do not necessarily produce greater benefits in SBM 
— are therefore consistent with Grossman et al. (2012). Since Tables 6 and 7 report the results 
of OLS regressions, we take steps to address possible omitted variable bias even though the 
results of the previous section suggest such steps are unnecessary. While we do not have an 
adequate number of instruments to produce 2SLS estimates, when we add the teacher relation- 
ship variable to the specifications to control for variation in student relationship ability, we find 
results similar to those reported in the tables. 


Facilitating Relationships in School-Based Mentoring 

The evidence compiled in the previous sections suggests that a close relationship between 
mentor and protege may be the key to the effectiveness of SBM. This section explores the 
characteristics of programs and mentors that may increase the likelihood of a close relationship. 
The goal is to help administrators leam how better to design SBM programs so as to facilitate 
relationship formation. 

Table 8 reports evidence on the program features and mentor characteristics associated 
with close mentor-protege relationships. Using the sample of all students assigned to treatment, 
we employ logistic regression to identify the factors influencing the likelihood that a protege felt 
close to his or her mentor, enriching the findings reported by Herrera et al. (2007). Column (1) 
presents odds ratios from our main specification. We found several statistically significant 
associations, but program type (academically focused or relationship-only) was not one of them. 
A program with an academic focus did not help or hinder relationship fonnation. Other program 
features appeared to be very consequential: proteges less frequently reported close connection 
with their mentors in programs where all pairs met at the same time and location and in those 
where pairs met only once or twice a month. Column (2) further investigates the negative effect 
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Table 7 


Estimated Impact of Match Length, Controlling for Relationship Closeness 



Overall academic performance 

Quality of work 

Completion of schoolwork 

Scholastic efficacy 

Short 

0.00 

0.06 

0.13 

0.04 


(0.09) 

(0.08) 

(0.11) 

(0.06) 

Medium 

0.13* 

0.13** 

0.14* 

0.06 


(0.07) 

(0.06) 

(0.08) 

(0.05) 

Long 

0.14** 

0.13** 

0.16*** 

0.09** 


(0.06) 

(0.05) 

(0.06) 

(0.04) 

Not close/short 

-0.08 

-0.10 

-0.03 

0.03 


(0.19) 

(0.17) 

(0.17) 

(0.10) 

Not close/medium 

0.14 

0.04 

0.14 

0.00 


(0.13) 

(0.13) 

(0.16) 

(0.10) 

Not close/long 

0.01 

-0.00 

0.04 

-0.05 


(0.14) 

(0.12) 

(0.10) 

(0.09) 

Close/short 

0.06 

0.15* 

0.26* 

0.04 


(0.10) 

(0.09) 

(0.14) 

(0.08) 

Close/medium 

0.11* 

0.12* 

0.15* 

0.08 


(0.07) 

(0.07) 

(0.09) 

(0.05) 

Close/long 

0 

0.15*** 

0.18*** 

0.10** 


(0.08) 

(0.06) 

(0.06) 

(0.04) 

R- 

0.534 0.537 

0.478 0.481 

0.385 0.387 

0.269 0.270 


NOTES: This table reports impact coefficients representing estimated absolute changes in the outcome measures. See Section III, “Outcome 
Measures and Key Variables,” for an explanation of how outcome measures were derived. Standard errors are shown in parentheses. Statistical 
significance levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 



of all matches meeting at the same time and location and finds that the large-group setting 
impaired relationship fonnation whether or not the mentor was a high school student. 

Proteges were less likely to feel close to college student mentors than to adult volun- 
teers, but more likely to feel close to high school student mentors. Same-race and same-gender 
matches were not more likely to be close. As column (3) shows, mentors who showed up when 
they were supposed to, as rated by the proteges, were significantly more likely to be in close 
relationships, especially when the program’s meetings were infrequent. (The possibility of 
endogeneity bias does exist though, as a mentor may not have shown up if the pair was not 
feeling particularly close.) Lastly, mentors who received more training before and during their 
mentoring assignments were more likely to have close relationships with their proteges. 

Although we have data on several other program features discussed in the literature 
(such as parent involvement and the number of contacts between the mentor and the case 
manager), we did not estimate the associations with variables not directly controlled by program 
administrators or those we believed were likely to be too endogenous. The factors shown in 
Table 8 are ones that programs can control and that may make it more likely for pairs to fonn 
close relationships. 
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Table 8 


Predictors of a Close Relationship (Odds Ratios) 



(1) 

(2) 

(3) 

Program features 

Academic focus 

0.69 

0.70 

0.24 


(0.47) 

(0.46) 

(0.23) 

All matches meeting at same time and location 
if high school student mentor 
if not high school student mentor 

0.31*** 

(0.13) 

0.24*** 

(0.13) 

0.40** 

(0.18) 

0.20*** 

(0.12) 

Some structured time 

1.35 

1.33 

1.25 


(0.59) 

(0.58) 

(0.83) 

Infrequent meetings 

0.28*** 

q 29*** 

0.00*** 


(0.13) 

(0.13) 

(0.00) 

Volunteer recognition 

1.01 

1.04 

1.98* 


(0.15) 

(0.16) 

(0.70) 

Age of student 

0.82*** 

0.82*** 

0 


(0.05) 

(0.05) 

(0.09) 

Mentor characteristics 

High school student 

1.28 

1.66 

1.96 


(0.45) 

(0.74) 

(1.11) 

College student 

0 27 *** 

q 27*** 

0.47 


(0.12) 

(0.12) 

(0.46) 

Same gender as student 

0.88 

0.88 

1.82 


(0.37) 

(0.37) 

(1.24) 

Same race as student 

1.15 

1.14 

1.39 

Mentor shows up 

Mentor shows up/infrequent program meetings 
Amount of training mentor received 

(0.33) 

(0.33) 

(0.79) 

1.80*** 

(0.30) 

3.17** 

(1.71) 

y 24 *** 
(0.12) 

Sample size 

474 

474 

271 


NOTES: This table presents the odds ratios associated with various program and mentor characteristics. 
Column (1) is our main specification. Column (2) allows the effect of the matches meeting at the same 
time and location to vary across mentor student status. Column (3) includes additional mentor characteris- 
tics. Specifications include site fixed effects and youth characteristics (for example, gender, race, or 
eligibility for free lunch). Cluster-robust standard errors are shown in parentheses. Statistical significance 
levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 
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V. Summary and Discussion 


This study finds evidence that unless a mentorship pair forms a relationship that the protege 
would rate as “somewhat close” or better, SBM program participation has no effect on a 
student’s end-of-school-year academic outcomes. We thus posit that this close relationship is 
the “active ingredient” of mentoring in this setting. This conclusion is very much in line with 
much of the theoretical work done on mentoring (Rhodes, 2005) and suggests that BBBSA’s 
school-based mentoring does not work merely as a tutoring program or by providing a context 
for specific training, as Cavell and Elledge suggest (forthcoming). We also find that SBM 
programs that focused more heavily on academic activity had no larger academic effects than 
those where mentors engaged primarily in social activities. 

Much of the field’s thinking about mentoring developed from community-based men- 
toring programs. That common wisdom says it takes time, often months, for a relationship to 
“gel,” for trust to be built so that the match can affect the life of a child (Grossman & Rhodes, 
2002). This study challenges this assumption for mentoring in the school setting by showing 
that a majority of proteges reported fairly close relationships even if they had been matched 
only for a month or two, and that such relationships accounted for positive impacts on academ- 
ic outcomes. Perhaps the school setting encourages children to trust these unrelated adults 
more quickly, or children may have lower thresholds for close relationships with “school 
adults” than with “community adults.” The fact that relationships between proteges and 
mentors in the BBBSA SBM program seem to gel fairly quickly may be why in their analyses 
with these same data Grossman et al. (2012) find little relationship between months of mentor- 
ing and academic impacts. 

Program staff and empirical researchers alike often worry that the children able to form 
the best relationships are those who would have normally done well anyway — in school or life 
— because of hard-to-measure characteristics such as persistence, or a child’s ability to get 
along with adults. But we find little evidence for this type of sorting among proteges with close 
relationships. If this type of process were occurring, we would see large differences between the 
2SLS impact estimates (which account for this potential self-selection) and the estimates 
obtained from the simpler OLS regressions. From a technical point of view, this finding is good 
news because it means we can control for closeness directly, without having to worry that this 
indicator is correlated with the error term. But from a programmatic point of view the finding is 
even more important, as it suggests that BBBSA SBM programs can be similarly helpful to all 
types of students. 

Our estimated treatment-on-treated effect sizes, ranging from 0.13 to 0.18 standard de- 
viations, compare favorably to the changes in the outcome measures for the control students 
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over the course of the year. For control students, overall academic performance improved by 
0.054 (standard deviation = 0.814) and quality of work by 0.016 (standard deviation = 0.729), 
while completion of work decreased by 0.038 (standard deviation = 0.873) and scholastic 
efficacy by 0.034 (standard deviation = 0.612). Moreover, the estimated effects of a close 
relationship with a mentor assigned through the BBBSA SBM program are comparable to the 
size of the impacts of many other school-based interventions on academic outcomes. For 
example, in Project STAR, the academic achievement impacts of reduced class sizes were an 
estimated 0.1 to 0.2 standard deviations (Krueger, 1999). Similarly, a meta-analysis of after- 
school programs reported average effect sizes ranging from 0.05 standard deviations on reading 
to 0.17 on math skills (Lauer, 2006). Another meta-analysis of explicitly nonacademic after- 
school programs (Durlak & Weissberg, 2007) found impacts on grades ranging from 0.24 
standard deviations for programs using evidence-based approaches to 0.05 for those not 
incorporating best practices. 

This study also sheds light on the perplexing rematch result reported by Grossman et al. 
(2012) in analyses of the same data analyzed in this paper. That study found that even though 
rematched proteges had slightly more total weeks of mentoring than proteges in intact matches, 
they performed less well than students who never had a mentor. The authors had no explanation 
for the result, but hypothesized that perhaps the relationships had not gelled sufficiently. That 
explanation does not seem plausible now that we know how quickly school-based mentoring 
relationships can click, at least from the protege’s perspective. 

Indeed, once we control for the quality of the matches, we see that although close, intact 
matches have the largest — and statistically most significant — estimated impacts on academic 
outcomes, rematched students in close relationships do not appear to be hurt. In all cases the 
estimated impacts are positive and fairly similar to those of the students who remained in their 
original mentoring relationships, though the standard errors on these coefficients are large. The 
negative effects appear to be centered among the rematched proteges who do not feel close to 
their new mentors (though because the subsamples are small, these negative coefficients are not 
statistically significant). These findings indicate that the emotional tenor of the relationship in 
the new match is the key issue programs should address. It could be that the second matches 
were rushed and less well suited to the students than their original matches. It could be that 
proteges felt rejected when their first mentoring relationships ended, affecting their ability to 
feel close to another program-assigned mentor quickly. More research should be done to 
investigate these issues. 

Similarly, this paper investigates the “length plus strength” hypothesis that grew out of 
earlier work on the BBBSA CBM program, in which estimated impacts were observed to 
increase over time once it appeared a trusting relationship had formed (Grossman & Rhodes, 
2002). We find that only close relationships have impacts and that match length is not as im- 
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portant in the SBM programs we study. We do not use the measure of length that has been used 
in other match-length studies (the total number of weeks a protege meets with any mentor), as we 
are interested in understanding whether the impact a particular mentor has on a protege increases 
with the length of their relationship. We do find evidence that long and close relationships led to 
improvements in academic outcomes, but it is difficult to conclude definitively that shorter but 
close relationships had smaller or no positive effects. Positive impacts on the completion and 
quality of the student’s homework appear to happen rather quickly, while improvements in the 
teacher’s overall assessment of the student’s perfomiance and in the student’s assessment of his 
or her own scholastic competence do appear to grow larger with longer relationships. More 
research involving large numbers of students is needed to answer this question. 

The analyses carried out in this study suggest that the BBBSA SBM program (and po- 
tentially others with similar characteristics) can make close relationships more likely by incor- 
porating several program practices: 

• Not having pairs meet in the same area at the same time 

• Giving mentors structured activities to do with their proteges 

• Asking mentors to come at least three times a month 

• Providing BBBSA staff support to mentors 

It might be a challenge for underfunded schools to provide adequate mentoring space, 
especially if mentors and proteges need multiple locations to meet on their own: schools might 
have to pay custodians more to clean the spaces after hours, for example. Schools might be able 
to reduce costs by encouraging pairs to meet during students’ lunch, recess, or study hall time. 

Although we did not find that academically focused mentoring led to better academic 
outcomes if the mentoring relationship was not close, it still may be justified to center well- 
designed volunteer programs on academic help. The mentors studied here were given no 
instruction in tutoring, nor did they have a structured curriculum to follow. The evidence from 
this study should therefore not be used to conclude that volunteer tutoring programs are ineffec- 
tive. But greater attention to the importance of closeness could help inform the design of 
tutoring programs and other after-school programs. Program designers would do well to 
incorporate practices that make it easier for staff members and volunteers to form close relation- 
ships with the young participants and thereby potentially strengthen program effects. 

This study develops an empirical framework to investigate rigorously the mechanisms 
by which a mentoring program may improve academic outcomes. Given that we find the 
mentor-protege relationship to be the core element of mentoring, we believe that any analysis of 
experimental data on mentoring programs should consider the partial compliance model used in 
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this paper. Although instrumental variables and other nonexperimental techniques do not 
produce results as convincing as evidence from randomized control trials, in the end it is not 
possible to randomly assign close relationships. Statistical techniques can appropriately adjust 
for the presence or absence of closeness. Building on the results of this study, future experi- 
ments can include designs created to refine our understanding of the mechanism behind mentor- 
ing effectiveness, with randomly selected mentoring matches or programs encouraged to 
incorporate either more relationship-building activities or more academically focused ones. 

Many schools are struggling to meet the standards set by the No Child Left Behind Act 
of 2001, and the future holds even more mandates and struggles as states implement the 
Common Core Standards. Under binding budget constraints, volunteer labor will be increasing- 
ly attractive. We find that school-based mentoring programs that aim to provide relationships 
can improve academic outcomes. Not every adult can be a good tutor, but many of them could 
help students academically by being a caring presence. 
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MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



